VIDEO INDEXING USING HIGH RESOLUTION STILL IMAGES 

1 Technical Field 

2 The technical field relates to video imaging system, and, in particular, to video 

3 indexing system. 

4 Background 

5 Users are increasingly using video cameras to record home videos, television 

6 programs, movies, concerts, or sports events on a disk or DVD for later or repeated 

7 viewing. A video camera typically records both video and audio to generate a video 

8 sequence, which can be stored in a secondary storage, such as a hard disk or a CD-ROM. 

9 Such video sequences typically have varied content or great length. Since a user 

10 normally cannot write down what is on a video sequence or where on a video sequence 

1 1 particular scenes, movies, events are recorded, the user may have to sit and view an entire 

12 video sequence to remember what was recorded or to retrieve a particular scene. Video 

13 indexing allows a user to have easy access to different sections of the video sequence so 

14 that the user do not need to fast forward through the whole video sequence. 

1 5 Current video indexing devices use video content analysis that automatically or 

16 semi-automatically extracts structure and meaning from visual cues in a video. After, for 

17 example, a video clip is taken from a television (TV) program or a home video, a 

1 8 computer will generate particular indexes so that a user can jump to a particular section in 

1 9 the video sequence. 

20 However, automatic video indexing typically generates different key frames, with 

21 the need for extensive post-processing, which involves automatic search for shot changes, 

22 scene changes, and ultimately, frames that may serve as key-frames. In addition, 

23 automatic video indexing may or may not help a user find a particular video event within 

24 a recording. 

25 Summary 

26 A method for video indexing using still images includes acquiring still images 

27 during a video sequence recording using an image/video acquisition device, processing 

28 and transmitting the video sequence and the still images by the image/video acquisition 

29 device, and indexing the video sequence using the still images, so that a user can 

30 selectively view the video sequence using the still images as video indices. Since the still 

3 1 images used to index video files are acquired during the video sequence recording, the 
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1 user can jump directly to a position within the video files where a particular still image is 

2 acquired without the need to fast forward or rewind. 

3 The still images typically have higher resolution than the video sequence. By 

4 selecting the high resolution still images to capture memorable moments, the user is able 

5 to easily index the video sequence using the most memorable still image pictures and to 

6 later view the video sequence around the instances of the still images. 

7 Description of the Drawings 

8 The preferred embodiments of the method for video indexing using high 

9 resolution still images will be described in detail with reference to the following figures, 

10 in which like numerals refer to like elements, and wherein: 

1 1 Figure 1 illustrates an exemplary joint video and still image pipeline; 

12 Figure 2 illustrates an exemplary method for video indexing using high resolution 

13 still images; 

14 Figure 3 illustrates an exemplary image/video acquisition device that processes, 

15 transmits, and stores video sequences and high resolution still images in parallel; 

16 Figure 4 illustrates an exemplary hardware components of a computer that may be 

17 used to in connection with the exemplary method of Figure 2 for video indexing using 

1 8 high resolution still images; and 

1 9 Figure 5 is a flow chart illustrating the exemplary method of Figure 2 for video 

20 indexing using high resolution still images. 

21 Detailed Description 

22 Using a image/video acquisition device, a user may acquire still image pictures at 

23 the same time as a video sequence recording. The still images typically capture 

24 memorable moments and may be used to index the video sequence so that a user may 

25 selectively view the video sequence using the still images as video indices. The video 

26 indices are similar to chapters within digital video disc (DVD) movies that enable a user 

27 to jump directly to a particular chapter without having to fast forward. 

28 Figure 1 illustrates an exemplary joint video and still image pipeline. The 

29 exemplary joint video and still image pipeline is capable of delivering a video sequence 

30 120 and still images 1 10 at the same time. For example, while the video sequence 120 is 

3 1 being recorded, a snapshot 102 may be taken to generate a still image 110. The 

32 processing of the video sequence 120 and the still image 1 10 may be in parallel. The 

33 joint video and still image pipeline is described, for example, in U.S. Patent Application, 

34 entitled "Concurrent Dual Pipeline for Acquisition, Processing and Transmission of 
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1 Digital Video and High Resolution Digital Still Photographs/' filed on the same day 

2 herewith, which is incorporated herein by reference. 

3 The joint video and still image pipeline leverages existing processing from 

4 cameras, i.e., demosaicing, color processing, and image compression. Additionally, the 

5 joint video and still image pipeline takes advantage of existing expertise on digital video, 

6 i.e., video compression and video streaming and transcoding. The still images 110 

7 typically have high resolution with, for example, 2-4 mega pixels. The high resolution 

8 still images 110 also have sophisticated demosaicing that leaves almost no demosaicing 

9 artifacts and high quality color correction that generates accurate color. On the other 

10 hand, the video sequence 120 typically has mid or low resolution, for example, 640 x 480 

1 1 resolution. The high resolution still images 110 also have sophisticated demosaicing that 

12 leaves almost no demosaicing artifacts and high quality color correction that generates 

13 accurate color. On the other hand, the videos 120 typically have mid or low resolution 

14 with, for example, 640 x 480 resolution. In contrast to high resolution still images 110, 

1 5 the mid or low resolution videos 120 have fast demosaicing and fast color correction, 

1 6 which produces high frame rate. The video is then compressed and streamed with low 

17 delay and good error protection. 

18 Figure 2 illustrates an exemplary method for video indexing using high resolution 

19 still images 110. During a video sequence 120 recording, high resolution still images 1 10 

20 may be acquired by a user, for example, by pressing a snapshot button 348 (shown in 

21 Figure 3) on an image/video acquisition device 300, such as a joint video and still image 

22 pipeline camera (shown in Figure 3). The high resolution still images 110 may be 

23 considered as key frames in the video sequence 120, and may be used to index the video 

24 sequence 120. The user may later selectively view the video sequence 120 using the high 

25 resolution still images 1 10 as video indices. Since the high resolution still images 110 

26 typically capture the most memorable moments, linking the most memorable images to 

27 points in time within the video sequence 120 enables the user to relive memorable 

28 experiences around the high resolution still images 110. 

29 Video indexing is described, for example, in "Content-Based Browsing of Video 

30 Sequences" by Arman et al., ACM multimedia, pages 97-103, 1994; and "Content Based 

3 1 Video Indexing and Retrieval" by Smoliar et al., IEEE multimedia, pages 62-72, 1994, 

32 which are incorporated herein by reference. Arman et al. disclose a novel methodology to 

33 represent the contents of a video sequence. The methodology uses a content-based 

34 browsing system that forms an abstraction to represent each shot of the sequence by using 
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1 a representative frame, and allows a user to easily navigate the frame, i.e., rapidly view a 

2 video sequence in order to find a particular point within the sequence. Smoliar et al. 

3 disclose a method for content-based video indexing and retrieval. The method includes 

4 parsing the video stream into generic clips, indexing the video clips when inserted into a 
, 5 database, and retrieval and browsing the database through queries based on text and/or 

6 visual examples. 

7 For example, during a birthday party, a parent may snapshot a high resolution still 

8 image 1 1 0(a) of a child blowing candles without interrupting a video sequence 120 

9 recording. Later, while still recording the birthday party, the parent may snapshot again 

10 to capture a high resolution still image 1 10(b) of kids playing games, or 1 10(c) of kids 

1 1 singing, and so on. The high resolution still images 110 acquired typically represent the 

12 most memorable moments of the event. The family may later use the high resolution still 

13 images 1 10 as video indexing and proceed directly to the most memorable moments in 

14 the video sequence 120. 

15 Figure 3 illustrates an exemplary image/video acquisition device 300, such as a 

16 joint video and still image pipeline camera, that processes, transmits, and stores video 

17 sequences 120 and high resolution still images 1 10 in parallel. The camera 300 includes a 

18 sensor 340, a still image pipeline 310, a video pipeline 320, and a local storage 350. The 

1 9 still image pipeline 3 1 0 and the video pipeline 320 may be located on the camera's 

20 hardware/firmware, application specific integrated circuits (ASICs), microprocessor 

21 and/or digital signal processor. The local storage 350 may be a solid state memory, 

22 which is similar to SDMemory cards from Panasonic, or a microdrive, which is similar to 

23 microdrives hard drives from IBM. Using, for example, a snapshot button 348, the sensor 

24 340 of the camera 300 may acquire high resolution still images 1 10 at the same time as 

25 recording a video sequence 120. Other types of user interaction, such as voice command 

26 or touch screen liquid crystal display (LCD), may be used to acquired the high resolution 

27 still images 1 10 during the video sequence recording. Next, the video sequence 120 and 

28 the still images 1 10 may be processed in parallel using the joint video and still image 

29 pipeline, 320 and 310, respectively. Thereafter, the video sequence 120 and the still 

30 images 110 may be transmitted and/or stored in the local storage 350 on the camera 300. 

3 1 Alternatively, the video sequence 120 and the still images 1 10 may be stored in a remote 

32 storage on a server/computer, such as a hard disk, a CD-ROM, or a server connected to a 

33 network. The high resolution still images 110 may be labeled, for example, as image #1, 

34 image #2, or image #3, within the video sequence 120 for easy indexing. 
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1 Figure 4 illustrates an exemplary hardware components of a computer 400 that 

2 may be used to in connection with the exemplary method for video indexing using high 

3 resolution still images 1 1 0. The computer 400 has a connection with a network 41 8, such 

4 as the Internet or other type of computer or telephone networks, for sending recorded 

5 video sequence 120 and still images 1 10 to friends and family by, for example, email. 

6 The computer 400 typically includes a memory 402, a secondary storage device 412, a 

7 processor 414, an input device 416, a display device 410, and an output device 408. 

8 The memory 402 may include random access memory (RAM) or similar types of 

9 memory. The secondary storage device 412 may include a hard disk drive, floppy disk 

10 drive, CD-ROM drive, or other types of non-volatile data storage. The secondary storage 

1 1 device 412 may correspond with various databases or other resources. The processor 414 

12 may execute applications or other information stored in the memory 402, the secondary 

1 3 storage 4 1 2, or received from the Internet or other network 418. The input device 416 

14 may include any device for entering data into the computer 400, such as a keyboard, key 

15 pad, cursor-control device, touch-screen (possibly with a stylus), or microphone. The 

16 display device 410 may include any type of device for presenting visual image, such as, 

17 for example, a computer monitor, flat-screen display, or display panel. The output device 

18 408 may include any type of device for presenting data in hard copy format, such as a 

1 9 printer, and other types of output devices including speakers or any device for providing 

20 data in audio form. The computer 400 can possibly include multiple input devices, output 

21 devices, and display devices. 

22 Although the computer 400 is depicted with various components, one skilled in 

23 the art will appreciate that this computer can contain additional or different components. 

24 In addition, although aspects of an implementation consistent with the present invention 

25 are described as being stored in memory, one skilled in the art will appreciate that these 

26 aspects can also be stored on or read from other types of computer program products or 

27 computer-readable media, such as secondary storage devices, including hard disks, floppy 

28 disks, or CD-ROM; a carrier wave from the Internet or other network; or other forms of 

29 RAM or ROM. The computer-readable media may include instructions for controlling 

30 the computer 400 to perform a particular method. 

3 1 After the video sequence 120 and the high resolution still images 1 10 are acquired 

32 by the camera 300, the video sequence 120 and the high resolution still images 1 10 may 

33 be downloaded to a computer 400 either by transmitting over wireless channels or 

34 through a wired connection, such as universal serial bus (USB) or Firewire (IEEE 1394). 
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1 Alternatively, the computer 400 may read the local storage 350 of the camera 300 by 

2 directly connecting to a reader of the computer 400. After downloading the recorded 

3 video sequence 120 and the high resolution still images 1 1 0, the video sequence 120 may 

4 be played back either on a liquid crystal display (LCD) (not shown) of the camera 300 or 

5 on a display device 410 of the computer 400. The LCD or the display device 410 may 

6 display the high resolution still images 1 10 as labeled icons, for example, image #1, 

7 image #2, or image #3. A particular high resolution still image 110, for example, image 

8 #1, may be displayed by clicking on an associated icon as displayed on the display device 

9 410. 

10 In addition, the high resolution still images 110 may be used to index the video 

1 1 sequence 120. For example, when viewing the recorded video sequence 120 using the 

12 computer 400, a user may double click on one of the icons, for example, image #2, and 

13 start viewing the video sequence 120 from a point in time associated with the high 

14 resolution still image #2. By linking the most memorable images to points in time within 

15 the video sequence 120, the user may easily index the video sequence 120 using the most 

16 memorable still image pictures 1 1 0. 

17 The video sequence 120 and the high resolution still images 1 1 0 may also be 

1 8 saved on a server connected to the network 41 8, to be retrieved by other users. 

19 Alternatively, the video sequence 120 and the high resolution still images 110 may be 

20 transmitted to other users through the network 418 by, for example, e-mail. A friend or a 

21 family member who receives the video sequence 120 and the still image 110 may then 

22 selectively view the video sequence 120 using the high resolution still images 1 10 as 

23 video indices. 

24 Figure 5 is a flow chart illustrating the exemplary method for video indexing 

25 using still images. An image/video acquisition device 300, such as a joint video and still 

26 image pipeline camera, enables a user to acquire still image pictures 1 10, which are 

27 typically high resolution still images, during a video sequence 120 recording, step 510. 

28 The still images 1 10 may be acquired during the video sequence 120 recording using, for 

29 example, a snapshot button 348, on the camera 300, step 520. Next, the video sequence 

30 120 and the high resolution still images 110 may be processed using a joint video and still 

3 1 image pipeline, step 530. Both the video sequence 120 and the high resolution still 

32 images 110 may be transmitted and stored in a local storage 350 or a remote storage, step 

33 540, where the high resolution still images 110 may be used to index the video sequence 

34 120, step 550. 
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1 A computer may be used to selectively view the video sequence 120 using the 

2 high resolution still images 1 10 as video indices, step 560. Alternatively, the user may 

3 choose to view the video sequence 120 on the camera's LCD and use the still images 110 

4 as video indices. The user may click on one of the high resolution still images 1 10 to 

5 start viewing the video sequence 120 from a point in time associated with that high 

6 resolution still image 1 10, step 570. The user may also print the high resolution still 

7 images 1 10, step 580. In addition, the video sequence 120 and the high resolution still 

8 images 110 may be sent through a network 418 to other users, such as friends and family, 

9 so that the other users may also selectively view the video sequence 120 using the high 
10 resolution still images 1 10 as video indices, step 590. 

i^L 1 1 While the method and apparatus for video indexing using high resolution still 

y 12 images have been described in connection with an exemplary embodiment, those skilled 

f t: 13 in the art will understand that many modifications in light of these teachings are possible, 

p 14 and this application is intended to cover any variations thereof. 
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